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May  20, 1992 

The  Honorable  Sam  Nunn 
Chairmen  Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Les  Aspin 
Chairman,  Committee  on  Armed  Services 
House  of  Representatives 

The  Composite  Health  Care  System  (CHCS)  is  an  automated  medical 
information  system  being  developed  by  the  Department  of  Defense  for  use 
in  664  military  medical  facilities  worldwide.  The  system  is  intended  to 
improve  the  quality  of  care  and  operational  efficiency  of  these  facilities  by 
integrating  a  wide  array  of  data  needed  for  managing  and  treating  patients. 

Defense  is  currently  seeking  approval  to  deploy  CHCS  from  its  M^or 
Automated  Information  Systems  Review  Committee  (MAISRC).  If  MAISRC 
approves  CHCS  for  deployment,  Defense  will  submit  the  results  of  the 
operational  test  and  evaluation  (OT&E)  and  a  cost/benefit  analysis,  as 
required  b\  law,  to  your  committees  before  awarding  the  full-deployment 
contract.  Defense  plans  to  obtain  a  maisrc  decision  in  May  1992. 

This  report  is  intended  to  update  you  on  Defense’s  progress  in  developing 
and  testing  CHCS.  Our  objectives  were  to  ( 1 )  determine  the  status  of  critical 
system-development  issues;  (2)  assess  the  OT&E  process  and  results,  to 
date;  and  (3)  review  the  system’s  cost/benefit  analysis.  Details  of  our 
objectives,  scope,  and  methodology  are  found  in  appendix  I. 


Results  in  Brief 


CHCS  is  not  ready  to  be  deployed.  l\vo  critical  system-development  and 
operational  issues  remain  unresolved— ntultiple  patient  records  and 
archiving  patient  records.  The  test  sites  iiave  identified  a  problem  with 
multiple  patient  records  which,  until  resolved,  thresitens  patient  safety 
since  physicians  could  provide  improper  care  on  the  basis  of  incomplete 
patient  medical  information.  The  test  sites  have  also  idehlftied  a  problem 
caused,  primarily,  by  the  lack  of  capabil’ty  to  archive  patient  daUi. 
Specifically,  most  test  sites  have  indicated  that  .system  performance  suffers 
as  a  result  of  the  volume  of  data  the  system  must  handle— an  outgrowth  of 
the  lack  of  archiving  capability.  This  peiformance  problem  is  most  i)ften 
visible  in  the  form  of  slow  response  tin\es  for  clinical  users.  These  two 
critical  issues  affect  the  tunent  deployment  decision.  In.addition,  Defense 
has  made  limited  progress  on  an  efficient  method  of  entering  ph.\.^K  i.uis' 


Cage  1 


('.AO,i.MTEC-i»2-51  CHCS  .Not  Keail>  l  o  Ho  Depliijed 


B-220732 


inpatient  orders.  The  delay  stems  from  the  current  design,  which 
physicians  find  to  be  not  user  friendly  and  contrary  to  their  usual  work 
processes.  This  problem  could  have  a  significant  impact  on  CHCS 
deployment. 

Operational  testing  was  supposed  to  provide  Defense  with  sufficient  data 
for  making  an  informed  decision  on  whether  the  system  is  ready  to  be 
deployed.  However,  test  results  are  inconclusive  because  management 
limited  the  number  of  test  sites  included  in  OT&E,  failed  to  conduct  tests  in 
realistic  environments,  and  failed  to  execute  parts  of  the  test  plan  properly. 
Defense  does  not,  therefore,  have  a  basis  for  determining  that  CHCS  is 
operationally  effective  and  suitable  for  deployment. 

The  CHCS  cost  estimate— when  applying  standard  Defense  life-cycle  cost 
directives— is  more  than  $400  million  above  the  $  1 .6-bil!ion  cap 
established  by  the  Congress.'  In  addition.  Defense  has  not  validated  a 
significant  portion  of  the  system’s  $2.3  billion  in  benefits,  and  it 
acknowledges  that  important  management  changes  will  have  to  be  made 
before  Defense  can  realize  them. 
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The  goal  of  CHCS  is  to  improve  the  care  of  patients  at  Defense  medical 
treatment  facilities.  The  system  is  supposed  to  provide  comprehensive, 
integrated  daUi  for  patient  management  and  treatment  in  a  timely  and 
easily  accessible  manner.  Defense  expects  the  degree  of  cues’  data 
integration  to  be  greater  than  currently  available  in  civilian  hospital 
information  systems.  Once  fully  deployed,  CHCS  would  replace  manual  and 
outdated  automated  information  systems  now  in  use. 

cues  is  intended  to  provide  physicians  with  immediate  access  to  all 
portions  of  a  patient’s  medical  record.  The  system  would  also  integrate 
nine  nnyor  hospital  functions  to  improve  patient  care  and  operational 
efficiency.  These  nine  functions  include  pharmacy,  laboratory,  radiology, 
nursing,  patient -appointment  scheduling,  patient  administration,  inpatient 
clinical  services,  outpatient  clinical  services,  and  other  ancillary  services. 

In  March  1988  Defense  awarded  a  contract  to  Science  Applications 
International  Corporation  (s.Mi  )  to  develop,  test,  deploy,  and  support 
(  IK'S.  Since  that  time,  SAK  has  developed  and  tested  the>sy.sieni  at  a 
luimher  of  sites.  However,  because  of  software  delay*,  tlie  time  needed  to 
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complete  the  required  testing  and  evaluation  has  been  extended  a  number 
of  times.  As  a  result,  a  decision  on  the  deployment  of  CHCS  has  not  yet  been 
made. 

Defense  has  obtained  support  from  congressional  conferees  to  take  a 
two-phased  approach  t."'  the  deployment  of  CHCS.  A  decision  on  whether  or 
not  to  deploy  the  first  phase  is  expected  in  May  1992.  This  deployment  is 
intended  to  include  all  CHCS  functions  listed  above  except  nursing, 
inpat'ent  clinical  services  (inpatient  order  entry),  and  parts  of  other 
ancillary  services.  Deployment  of  the  second  phase  is  scheduled  for  1994 
and  is  expected  to  include  the  remaining  CHCS  functions. 

The  Congress  capped  CHCS  life-cycle  costs  at  $  1 .6  billion.  By  the  end  of 
April  1992,  Defense  had  obligated  about  $515  million  (actual  dollars)  for 
the  development  and  deployment  of  the  system. 


Critical  Development 
Issues  Remain 
Unresolved 


The  first  phase  of  CHCS,  for  which  Defense  wants  to  obt  ain  approval  to 
deploy  this  month,  does  not  include  two  important  capo.bilities:  an  effective 
method  for  identifying,  eliminating,  and  preventing  multiple  patient 
records;  and  an  effective  method  for  archiving  and  retrieving  pr.tient 
records.  The  existence  of  multiple  patient  records  is  a  threat  to  c.  patient’s 
safety  because  it  can  result  in  a  physician’s  providing  treatmen',  >1;  the 
basis  of  incomplete  patient  medical  information.  The  capability'  tv'  archive 
and  retrieve  patient  records  is  essential  to  efilcient  system  opeiv;;  ons. 

The  capability  to  enter  physicians’  inpatient  orders  Ln  a  streamhried 
manner  is  now  scheduled  to  be  included  n  the  second  phase  0'  rncs 
software,  to  be  deployed  in  1994.  While  Defense  plans  an  initia  system 
deployment  without  this  capability,  it  remains  critical  io  physrc  an 
acceptance  of  CHCS  and  to  the  ability  of  the  system  to  effectively  and 
efficiently  support  the  inpatient  activities  0;  clinicians.  Defejnsi .  however, 
has  had  limited  success  in  developing  a  '-oiutinn  to  this  piobi.mi. 


Multiple  Patient  Records  Anus  user  sometimes  fails-for  a  vai  rely  of  reasons-to  1(»«  ate  a  record 

for  a  patient  and,  <ussuini;ig  the  patioi:!  has  no  previous  hit  .ory  at  the 
facility,  creates  an  additional  1  ^'coni  This  may  occur  beca.ise.  in  retrieving 
patient  data,  cites  allows  data  sea: dies  by  name.  In  such  searches,  name 
reque.sts  must  be  identical  to  tlie  mimes  on  patients’  existing 
records-iiu'Iudmg  punctuation,  s{  acing,  special  characters  (i.e.,  hypliens), 
sponsor  names,  and  abbreviations.  If  the  searched  names  are  not  found. 
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then  CHCS  allows  users  to  create  new  records.  Multiple  records  were  also 
created  when  data  bases  of  earlier  systems  that  contain  multiple  records 
were  used  during  data  conversions  to  establish  new  CHCS  patient  records. 

Physicians  who  attempt  to  call  up  a  patient’s  record  may  receive  any  one  of 
the  multiple  records  and  base  medical-care  decisions  on  incomplete  patient 
data.  This  situation  occurs  because  multiple  records  are  not  automatically 
linked  together  by  the  system.  Currently,  the  system  does  not  automatically 
alert  the  physician  that  critical  patient  information  may  exist  in  one  or 
more  additional  records  for  that  patient.  As  a  result,  decisions  on  the 
patient’s  treatment  could  be  made  that  Utay  have  serious  health 
consequences.  For  example,  a  patient’s  allergic  reaction  to  a  particular 
drug  may  be  in  only  one  of  the  multiple  records,  but  if  the  physician  calls 
up  another  record  when  treating  the  patient,  treatment  could  then  be  given 
without  knowledge  of  this  important  information. 

As  of  March  24, 1992,  Defense  estimated  that  the  data  bases  at  CHCS  test 
sites  contained  from  2.1  to  5.5  percent  multiple  patient  records.  To 
illustrate  the  significance  of  this  problem.  Defense  estimates  that  4.3 
percent-over  18,500-of  the  records  at  Tripler  Army  Medical  Center  in 
F  nolulu,  Hawaii,  are  multiples.  Defense  estimates  one  full-time  person 
can  oversee  the  actions  required  to  merge  five  multiple  records  per  day.  At 
this  rate— with  existing  technology— it  would  take  until  May  2003  to 
eliminate  all  multiple  records  at  Tripler. 

Defense  has  tried  to  reduce  the  magnitude  of  this  problem  by  (1) 
restricting  the  number  of  personnel  who  are  authorized  to  establish  new 
records,  and  (2)  purging  multiple  records  from  existing  systems  that  will 
be  used  to  establish  future  CHCS  data  bases.  In  addition,  by  September,  the 
contractor  plans  to  complete  field  testing  of  software  that  should  identify 
and  mark  multiple  records.  However,  until  these  management  actions  and 
software  changes  are  fielded  and  tested,  CHCS  users  will  continue  to  create 
multiple  patient  records  and,  potentially,  retrieve  and  work  from 
incomplete  patient  records. 


Archiving  Archiving  is  the  ability  to  store  patient  data  off-line  and  retrieve  it  as 

needed.  Although  essential  to  effective  CHCS  operation,  archiving  has  yet  to 
be  operationally  tested  and  evaluated.  Defense  has  developed  a  potential 
archiving  capability  but  hits  not  completed  field  testing  at  the  initial  test 
site. 
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Inpatient  Order  Entry 


Defense  has  identified  the  absence  of  archiving  as  having  a  negative  effect 
on  system  performance.  System  performance  directly  affects  not  only 
users’  satisfaction,  but  also  their  ability  to  use  other  CHCS  capabilities.  In  an 
April  1992  contractor  progress  report  for  CHCS,  8  of  the  12  primary  test 
sites  listed  system  performance  as  one  of  the  top-priority  problems 
requiring  resolution.  Two  sites— Walter  Reed  Army  Medical  Center  in 
Washington,  D.C.,  and  the  U.S.  Air  Force  Medical  Center  at  Keesler  Air 
Force  Base  in  Biloxi,  Mississippi— had  average  response  times  that  were  50 
percent  above  acceptable  performance  levels. 

Designing  and  implementing  an  effective  CHCS  archiving  capability  is 
complex.  It  is  technically  challenging,  in  part  because  CHCS  software  is 
highly  integrated,  relationships  among  patient  data  are  complex,  and  a 
nontraditional  file  structure  is  used.  The  extraction  of  patient  data  from 
this  file  structure  must  be  performed  in  a  flawless  manner  to  ensure  that 
relationships  between  all  segments  of  the  active  data  base  are  preserved 
and  that  the  archived  data  can  be  reassembled  later,  when  needed  for 
patient  care  or  medical  research.  Before  such  a  complex  and  important 
capability  is  fully  deployed,  it  is  essential  that  it  be  thoroughly  tested  in  a 
representative  set  of  hospital  environments. 

Deploying  CHCS  without  assurances  that  the  archiving  solution  can  be 
effectively  incorporated  into  the  system  could  ultimately  result  in  serious 
operational  problems.  Without  archiving,  system  response  time  slows  as 
the  volume  of  data  stored  on-line  reaches  disk-capacity  limits.  Efforts  to 
improve  the  response  time  can  be  time-consuming,  disruptive,  and 
eventually  ineffective,  as  even  more  data  are  stored  on  disk.  With  no 
effective  way  to  store  patient  data  off-line,  hardware  costs  will  continue  to 
increase  as  additional  disk  storage  and  related  equipment  (controllers  and 
faster  processors  with  more  input/output  channels)  are  added  to  handle 
increasing  volumes  of  data. 


Inpatient  order  entry  is  the  process  through  which  physicians  enter  the 
orders  for  the  treatment  of  hospitalized  patients.  As  we  reported  in 
September  1991,-  the  existing  inpatient  order-entry  capability  was 


■Meaic.il  ADI’  System:.  ChanKes  iii  Composite  Health  Care  System's  Deployment  StralejO’  Are  Cnwise 
(GAO/lMTEC-iH-17.  Sept.  30,  I'JiM ). 
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unacceptable  to  many  physicians,  primarily  because  of  the  length  of  time  it 
takes  physicians  to  enter  both  conditional  and  complex  orders®  in  contrast 
to  the  manual  system.  As  a  result  of  thio  resistance,  Defense  has  decided  to 
deploy  CHCS  without  the  inpatient  order-entry  capability,  and  to  conduct 
more  research  toward  development  of  an  acceptable  method  for  entering 
inpatient  orders.  Defense  has  performed  an  extensive  analysis  of  the 
inpatient  order-entry  problem  and  has  defined  a  set  of  potential  solutions. 
A  component  of  this  set  of  solutions  is  commonly  known  as 
one-line-order-entry.  This  component  will  allow  physicians  to  enter 
complex  and  conditional  orders  by  typing  a  minimal  amount  of  text. 

The  failure  to  develop  a  physicians’  inpatient  order-entry  system  will  have 
significant  cost  and  operational  implications  for  CHCS.  Until  physicians 
enter  their  own  orders,  other  hospital  staff  will  be  required  to  place  these 
orders  into  the  system  for  physicians.  This  requirement  will  not  only  add 
additional  cost  to  the  operation  of  CHCS,  but  will  also  make  the  system 
more  error-prone  because  someone  other  than  the  physician  will  be 
entering  orders. 

Defense  also  faces  a  mqjor  unknown  developmental  risk  in  the  designing 
and  testing  of  the  physicians’  inpatient  order-entry  capability.  The  required 
characteristics  of  the  physician  interface  for  inpatient  orders,  combined 
with  the  system  performance  requirements  and  characteristics  of  the 
computer  programming  language  used,  substantially  increase  the  difficulty 
and  risk  of  this  development  initiative.  It  is  still  uncertain  whether  an 
effective  and  efficient  inpatient  capability  can  be  designed. 

While  Defense  is  addressing  the  problems  associated  with  the  development 
of  a  physicians’  inpatient  order-entry  system,  the  Congress  directed  that 
alternative  solutions  for  inpatient  ordtr  entry  be  evaluated.  Consequently, 
in  February  1 992  Defense’s  CHCS  contractor  issued  a  request  for  proposals 
to  solicit  commercial  order-entry-system  solutions.  Evaluation  of  these 
submissions  is  expected  to  be  completed  this  month.  Defense’s  CHCS 
schedule  provides  for  the  installation  of  an  inpatient  order-entry  capability 
at  test  sites  in  1994.  However,  Defense’s  ability  to  meet  this  schedule  will, 
we  believe,  depend  heavily  on  the  quality  of  the  responses  to  the  request 
for  proposals  and  the  compatibility  of  the  solution  with  the  existing  CHCS 
software. 


('(iiKlilmiial  ordiT  is  a  (iriiceiltin.'  tlial  is  (lopciuloiU  upon  llie  outcome  of  a  prior  procedure.  For 
example  Take  vital  signs- if  temiierature  is  greater  tlian  100  degrees  f^ihrcnheit,  administer  'lyicnol." 
A  complex  order  involves  multiple  procedures  and  [lossibly  one  or  more  conditions. 
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Test  and  Evaluation 
Planning  and 
Implementation 
Inadequate 


The  primary  purpose  of  the  operational  test  and  evaluation  is  to  ensure 
that  only  operationally  effective  and  suitable  systems  are  delivered  to  the 
operating  forces.  OT&E  is  a  field  test-under  operationally  realistic 
conditions-to  determine  the  effectiveness  and  suitability  of  weapons, 
equipment,  or  munitions.  Testing  should  be  accomplished  in  an 
environment  as  operationally  realistic  as  possible.  Defense’s  Test  and 
Evaluation  Master  Plan  for  CHCS  states  that  for  test  results  to  be  valid,  the 
test  environment  of  the  selected  test  sites  should  be  representative  of  the 
environment  in  which  deployment  is  intended. 

OT&E  was  supposed  to  provide  useful  information  for  making  an  informed 
decision  on  whether  CHCS  was  ready  to  be  deployed.  For  the  data  to  be 
useful,  they  must  (1)  come  from  test  sites  that  are  representative  of  the 
tmiverse  of  Defense  medical  treatment  facilities,  and  (2)  include  results 
from  tests  of  all  software  installed  and  operational  at  the  test  sites. 
Inadequacies  in  OT&E  planning  and  implementation,  however,  produced 
results  that  are  inconclusive  and  not  representative  of  the  environment  in 
which  CHCS  is  to  be  deployed.  Therefore,  Defense  does  not  have  a  basis,  at 
this  time,  for  concluding  that  CHCS  is  operationally  effective  and  suitable 
for  deplojTnent. 

Defense  established  five  test  categories-deployment,  training, 
mobilization,  technical,  and  functional— to  determine  the  effectiveness  and 
suitability  of  CHCS.  Defense’s  Office  of  Health  Systems  Evaluation  was 
responsible  for  managing  the  OT&E  process.  Defense  contractors  collected 
and  summarized  test  data,  and  Defense  appointed  and  trained  an  ot&e 
analysis  team,  made  up  of  15  members  representing  all  three  military 
departments,  to  review  and  evaluate  the  test  data.  An  OT&E  report  was 
published  in  January  1992  identifying  system  strengths,  weaknesses,  and 
deployment  risks  in  each  of  the  five  test  categories. 

The  CHCS  program  manager  received  a  copy  of  the  OT&E  report  and  is 
developing  plans  to  address  the  system  weaknesses  identified.  We  also 
reviewed  the  OT&E  plans,  procedures,  and  results,  and  found  additional 
testing  inadequacies.  Overall,  OT&E  inadequacies  were  primarily  the  result 
of  (1)  weaknesses  in  Defense’s  overall  management  of  OT&E  and  (2) 
Defense’s  failure  to  conduct  some  tests  in  realistic  operating  environinents. 

Weaknesses  in  Defense’s  management  of  OT&E  were  especially  evident  in 
instances  in  which  the  number  of  test  sites  and  the  extent  of  testing  was  so 
limited  as  to  render  the  results  unusable  or  of  limited  use.  This  occurred 
because  Defense  made  changes  in  the  number  of  representative  test  sites. 
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Life-Cycle  Costs 
Exceed  Congressional 
Ceiling,  While  Benefits 
Remain  Largely 
Unsupported 


training  procedures,  and  overall  CHCS  deployment  strategy.  For  example, 
testing  of  the  CHCS  initial-training  and  deployment  capabilities  was 
conducted  at  only  one  site.  Further  diminishing  the  usefulness  of  the 
results  was  the  fact  that  this  site  had  activated  only  one  of  the  nine  CHCS 
functions— patient-appointment  scheduling.  Tests  conducted  under  such 
limited  conditions  are  not  representative  of  the  enviroiunent  in  which 
deployment  is  intended. 

Defense  also  conducted  tests  in  environments  that  were  not  operationally 
realistic.  For  example,  when  testing  the  system’s  technical  capability. 
Defense  ran  security  and  performance  tests  on  hospital  systems  with 
inactive  system-security  features.  The  test  sites  had  deactivated 
system-security  features  to  alleviate  response-time  problems.  Additionally, 
the  system-reliability  computations  did  not  include  the  component 
reliability  for  disk  drives.  Testing  and  analysis  under  such  limited 
conditions  renders  test  results  inconclusive.  Specifics  on  test  inadequacies 
and  the  related  background  are  included  in  appendbc  II. 

While  the  ot&e  test  plan  and  its  implementation  were  inadequate,  the  OT&E 
process  did  identify  17  fundamental  CHCS  management  weaknesses,  in  the 
areas  of  security,  training,  deployment,  capacity,  and  mobilization.  For 
example,  management  failed  to  establish  a  structured  security  program  to 
oversee  and  support  site-level  security  management.  As  a  result,  little  or  no 
training  was  provided  in  automated-data-processing  security,  audit-trail 
functions  were  turned  off  or  only  partially  employed,  and  contingency 
plans  were  often  nonexistent  or  unrealistic.  CHCS  program  management 
has  prepared  a  list  of  actions  it  expects  to  take  in  dealing  with  these 
problems. 


A  significant  problem  exists  with  respect  to  the  costs  and  benefits  of  CHCS. 
CHCS  life-cycle  costs-when  estimated  under  current  Defense  regulations 
and  guidance— exceed  the  congressionally  established  $  1.6-billion  ceiling 
by  more  than  $400  million.  In  addition,  Defense  has  had  a  great  deal  of 
difficulty  estimating  and  validating  CHCS  benefits.  The  most  recent  benefit 
estimate  of  $2.3  billion  is  based  on  questionable  data,  and  Defense 
acknowledges  that  many  of  the  benefits  cannot  be  realized  until  significant 
management  changes  are  implemented. 

Defense  regulations  require  that  prior  to  making  a  deployment  decision, 
appropriate  officials  must  ( 1 )  provide  a  life-cycle  schedule  with  realistic 
cost  and  acceptable  budget  estimates,  and  (2)  demonstrate  that  the  system 
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is  cost  effective  and  affordable  and  remains  the  best  alternative.  The 
program  cost  estimate  should  include  the  cost  of  all  resources  associated 
with  research  and  development,  investment,  and  operations  throughout  the 
system’s  economic  life.  The  benefit  analysis  should  identify  and  quantify 
actual  and  anticipated  benefits  that  reduce  cost  or  enhance  value  when 
compared  with  the  status  quo. 


Costs  Congress  increased  the  CHCS  cost  ceiling  from  $1.1  billion  to  $  1 .6  billion  in 

fiscal  year  1991.  The  current  CHCS  program  office  cost  estimate  creates  the 
illusion  that  total  costs  will  remain  within  the  ceiling  established  by  the 
Congress.  In  reality,  however,  the  total  cost  of  CHCS,  when  estimated  under 
current  Defense  guidance,  will  exceed  the  congressional  ceiling  by  more 
than  $400  million.  The  CHCS  program  office  cost  estimate  of  $1,538  billion 
does  not  follow  Defense  guidance,  requires  the  elimination  of  the  mid-life 
system  upgrade  that  the  program  office  used  to  justify  the  fiscal  year  1991 
cost-ceiling  increase,  and  excludes  all  operation  and  support  costs 
occurring  beyond  the  fifth  year  after  deployment  to  a  site. 

Guidance  from  the  Assistant  Secretary  of  Defense’s  Program  Analysis  and 
Evaluation  Office  states  that  Defense  components  should  estimate  system 
costs  using  a  standard  life  cycle  of  10  years  and  calls  for  a  system  upgrade 

4  to  6  years  after  a  system  reaches  initial  operational  capability.  Using  this 
guidance,  CHCS  costs  would  total  over  $2  billion,  or  more  than  $400  million 
over  the  congressional  ceiling. 

The  CHCS  program  office,  however,  estimates  CHCS  life-cycle  costs  at 
$1,538  billion.  The  CHCS  program  manager  defends  this  estimate  by 
characterizing  CHCS  as  a  “design-to-cost”  system  because  of  the 
congressional  ceiling.  This  means  that  since  the  Congress  has  limited  costs 
to  $  1 .0  billion.  Defense  is  developing  cost  estimates  that  stay  below  this 
ceiling.  This  process  has  resulted  in  the  (1)  elimination  of  the  mid-life 
system  upgrade,  and  (2)  institution  of  a  5-year,  “rolling-window”  concept. ' 
Under  this  concept,  CHCS  operation  and  support  costs  are  included  for  only 

5  years  after  deployment  to  a  site,  rather  than  the  10  years  prescribed  in 
Defense  guidance. 


'since  CHCS  cannot  be  deployed  to  all  locations  at  once,  the  program  office  is  using  a  rolling  window 
of  ,0  years  of  cumulative  costs  from  the  date  a  site  receives  Cl  ICS,  All  costs  beyond  the  5-year  iieriod 
are  ignored. 
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While  the  CHCS  program  office  continues  to  state  that  estimated  system 
costs  do  not  exceed  the  congressional  limits,  use  of  prescribed  Defense 
guidance  results  in  significant  increases  in  estimated  CHCS  costs.  In  effect, 
the  program  office  is  establishing  its  own  cost-estimating  guidance  in  order 
to  keep  costs  within  the  congressional  ceiling. 

Benefits  Wide  variations  continue  to  occur  in  both  the  amount  and  nature  of 

estimated  CHCS  benefits.  From  January  15, 1992,  to  April  3, 1992,  Defense 
conducted  five  benefit-estimate  studies  for  the  system.  The  estimated 
benefits  have  ranged  from  a  high  of  $3.8  billion  to  a  low  of  $1.7  billion. 

One  specific  benefit,  time  savings  from  electronic  mail,  accounted  for  $1.8 
billion  in  benefits  in  the  February  10  estimate,  but  was  less  than  one-half 
this  amount  in  the  latest,  April  3,  estimate.  The  following  figure  charts  the 
five  most  recent  benefit  estimates. 


Figure  1 :  CHCS  Benefit  Estimates 

Dollar  Savings  In  Billions 
4.0 


3.0 


2.0 


1.0 


0 

Jan  15  Ftbl  Fab  10  Feb  15  Apr  3 
1992  Estlmate.RsvIalon  Dates 


The  most  recent  Defense  estimate  indicates  that  CHCS  will  generate  about 
$2.3  billion  in  life-cycle  benefits.  Defense  expects  that  about  two-thirds,  or 
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$1.6  billion,  of  these  benefits  will  result  from  reduced  Civilian  Health  and 
Medical  Program  of  the  Uniformed  Services  (CHAMPUS)®  costs.  These 
CHAMPUS  benefits,  however,  are  not  supportable.  About  $832  million  of 
them  are  expected  to  result  from  the  use  of  electronic  mail.  Benefits  in  this 
category  come  from  time  saved  by  transmitting  information,  questions,  and 
messages  electronically.  Defense  expects  that  the  time  saved  by  physicians 
and  nurses  will  result  in  their  being  able  to  treat  more  patients.  Thus, 
CHAMPUS  costs  would  be  reduced  because  patients  who  now  receive  care 
from  civilian  medical  facilities  under  CHAMPUS  could,  instead,  be  treated  at 
a  military  medical  facility.  Defense,  however,  based  these  benefits  on 
information  provided  during  interviews  with  a  select  group  of  78 
phy.sicians  and  nurses  attending  a  CHCS  users  conference.  CHCS  officials 
agree  that  the  sample  used  for  this  estimate  is  inadequate,  and  they  are 
currently  expanding  this  sample  at  several  CHCS  sites. 

The  majority  of  the  remaining  CHAMPUS  benefits  are  based  on  improved 
records  maintenance  and  closer  scrutiny  of  the  need  for  prescribed  tests 
and  procedures.  These  benefits  were  based  on  results  from  studies  of  other 
automated  medical  systems,  some  performed  in  the  1970s.  Defense 
gathered  data  on  records  maintenance  improvements  during  OT&E,  but 
because  some  test  sites  already  had  some  degree  of  automation  and  made 
management  changes  during  testing.  Defense  was  unable  to  validate  for 
CHCS  the  benefits  identified  in  the  earlier  studies.  Similarly,  cost  reductions 
expected  from  closer  scrutiny  of  tests  and  procedures  are  based  on 
literature  research  rather  than  experience  with  CHCS. 

The  CHCS  program  manager  characterized  the  current  benefits  study  as  an 
indication  of  potential  benefits  and  stated  that  the  purpose  of  OT&E  does 
not  include  validation  of  benefits.  Further,  he  agreed  that  significant 
management  changes  would  be  necessary  before  these  projected  benefits 
could  be  realized.  The  CHCS  program  manager  is  currently  pursuing 
initiatives  which,  to  be  successful,  will  require  high-level  policy  changes. 
For  example:  (1)  hospital  commanders  could  implement  policies  that 
would  ensure  that  physicians  would  use  the  time  saved  from  the 
implementation  of  CHCS  to  add  CHAMPUS  patients  to  their  schedules,  and 
(2)  Defense  could  allocate  CHAMPUS  dollars  directly  to  hospital 
commanders— providing  them  with  the  incentive  to  manage  their  resources 
more  effectively  and  efficiently. 


^CHAMPUS  pays  for  health-care  costs  for  faniihes  of  uiiiforincd  services  members  who  are  m\able  to 
get  care  through  a  military  hospital  or  dime  ami  receive  medical  ser^-ices  from  private  providers. 
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Conclusions 


Recommendations 


CHCS  is  intended  to  enhance  Defense’s  ability  to  manage  patient  data  and 
improve  the  quality  of  care.  At  present,  however,  the  system  cannot 
adequately  accomplish  this  and  therefore  is  not  ready  for  deployment. 
Major  deficiencies  still  exist  in  the  syste.m’s  capabilities.  In  addition,  the 
scope  and  quality  of  system  testing  were  inadequate,  and  test  results  are 
inconclusive.  Until  Defense  corrects  these  deficiencies,  operational 
problems  will  continue  to  occur  and  patient  safety  may  be  threatened. 

Further,  increased  management  commitment  to  making  CHCS  an  integral 
part  of  medical-treatment-facility  operations  is  necessary  if  needed 
improvements  are  to  be  realized.  Many  corrective  actions  and  changes  in 
policy  are  planned  or  have  been  proposed  to  improve  CHCS 
management— particularly  in  the  areas  of  patient  safety,  benefits 
realization,  security,  and  training.  Until  these  actions  are  taken  and  the 
outcome  evaluated,  the  ability  of  CHCS  to  achieve  its  full  potential  is 
doubtful. 

A  significant  financial  investment  is  being  made  to  develop  a  hospital 
information  system  with  great  potential  benefits.  Given  its  importaiice  to 
the  well  being  of  service  men  and  women,  it  is  vital  that  the  system  meet 
established  development  requirements  in  order  to  help  ensure  that  it  is  safe 
for  use,  likely  to  be  deployed  successfully,  and  the  most  reasonable 
alternative  for  providing  needed  medical  support. 


In  order  to  help  ensure  the  success  of  CHCS  once  deployed,  we  recommend 
that  the  Secretary  of  Defense  direct  the  Assistant  Secretary  of  Defense  foe 
Health  Affairs  to  defer  approval  to  deploy  CHCS  until: 

•  the  ability  to  identify  and  remove  multiple  patient  records  has  been 
incorporated  into  the  software  version  of  CHCS  that  Defense  intends  to 
deploy  beyond  the  designated  test  sites, 

•  procedures  have  been  established  to  prevent  the  creation  of  multiple 
patient  records, 

•  the  capability  to  archive  and  retrieve  patient  data  has  been  successfully 
field  tested, 

•  a  sound  testing  methodology  has  been  developed  and  carried  out  for  those 
parts  of  the  OT&K  u'at  were  inadequate,  and 

•  a  complete  and  suppoitable  cost/benefit  analysis  has  been  performed. 

In  addition,  because  an  efficient  method  for  entering  physician  inpatient 
orders  is  significant  to  the  overall  success  of  (  lies,  we  also  recommend 
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[  that  the  Secretary  of  Defense  direct  the  Assistant  Secretary  of  Defense  for 

I  Health  Affairs  to  update  the  Senate  and  House  Committees  on  Armed 

Services  periodically  on  the  progress  being  made  on  the  development  of  a 
solution  to  the  inpatient  order-entry  problem. 


Agency  Comments 


Because  we  believe  that  it  is  important  for  maisrc  to  have  our  official 
position  regarding  the  deployment  of  CHCS,  we  requested  oral  comments 
on  a  draft  of  this  report.  We  provided  Defense  with  the  draft  report  on  May 
6, 1992,  and  requested  oral  comments  by  May  13, 1992.  On  May  14, 1992, 
Defense  indicated  that  although  it  intended  to  respond  to  the  report,  it 
could  not  provide  a  comprehensive  response  within  the  7  days  allotted. 
Further,  Defense  officials  stated  that  they  considered  the  issues  too 
significant  to  address  adequately  within  our  prescribed  time  frame. 

Although  we  did  not  obtain  oral  comments  on  our  draft  report,  we 
discussed  our  findings  and  recommendations  with  the  CHCS  Program 
Manager.  We  also  discussed  our  findings  with  the  Defense  officials  who 

(1)  were  responsible  for  the  oversight  of  all  CHCS  OT&E  activities, 

(2)  provided  the  independent  reviews  of  cost  and  benefit  data  for  the 
deployment  decision,  and  (3)  were  responsible  for  software  development. 
Defense  officials  generally  acknowledged  that  problems  exist  and  that 
these  problems  pose  deployment  risks.  However,  they  do  not  consider 
them  significant  enough  to  delay  a  deployment  decision.  Although  Defense 
is  taking  action  to  address  the  issues  discussed  in  our  report,  we  believe 
that  until  these  issues  are  resolved,  the  risk  of  making  a  deployment 
decision  at  this  time  is  too  great. 


We  are  sending  copies  of  this  report  to  the  Chairmen  of  the  House  and 
Senate  Committees  on  Appropriations;  the  Secretary  of  Defense;  and  the 
Director,  Office  of  Management  and  Budget.  Copies  will  also  will  be  made 
available  to  other  interested  parties  upon  request. 
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We  conducted  our  evaluation  from  July  1991  to  May  1992,  in  accordance 
with  generally  accepted  government  auditing  standards.  This  work  was 
performed  under  the  direction  of  Frank  W.  Reilly,  Director,  Human 
Resources  Information  Systems,  who  can  be  reached  at  (202)  512-6408. 
Other  major  contributors  are  listed  in  appendbc  III. 


Ralph  V.  Carlone 
Assistant  Comptroller  General 
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Objectives,  Scope,  and  Methodology 


The  National  Defense  Authorization  Act  for  Fiscal  Year  1987,  as  amended, 
requires  that  GAO  (1)  monitor  the  OT&E  phase  and  related  CHCS  acquisition 
activities,  and  (2)  submit  a  report  to  the  Senate  and  House  Committees  on 
Armed  Services  that  evaluates  OT&E  results  and  Defense’s  contract  award 
process  for  CHCS’  full  production  and  determines  whether  Defense 
conducted  OT&E  at  a  sufficient  number  of  sites  with  sufficient  software  in 
operation  to  warrant  a  full-deployment  decision.  The  act  requires  that  our 
final  report  on  OT&E  for  CHCS  be  issued  30  days  after  the  Senate  and  House 
Committees  on  Armed  Services  receive  Defense’s  report  on  the  OT&E 
results. 

Our  objectives  were  to  (1)  determine  the  status  of  critical 
system-development  issues;  (2)  assess  the  OT&E  process  and  results  to 
date;  and  (3)  review  the  system’s  cost/benefit  analysis.  In  conducting  our 
review,  we  examined  Defense’s  May  1991  Test  Evaluation  Master  Plan  and 
accompanying  Detailed  Test  and  Analysis  Plan;  reviewed  the  OT&E 
processes  and  procedures  Defense  and  its  contractors  followed  during 
testing  and  reporting;  verified  that  Defense  provided  full  disclosure  of 
OT&E  data  to  its  OT&E  Test  Analysis  Team,  as  well  as  to  the  surgeon  general 
of  each  military  department;  reviewed  the  CHCS  life-cycle  cost  analysis  and 
supporting  documentation;  reviewed  the  CHCS  benefits  reports  and 
supporting  documentation;  reviewed  Defense’s  most  current  (May  1992) 
CHCS  system  decision  paper  and  supporting  documentation;  evaluated  the 
montlily  progress  reports  provided  to  Defense  by  the  CHCS  contractor 
througii  April  1992;  and  trad.  ’  'll  delivery  orders,  including 
modifications,  that  Defense  issueu  CHCS  contract  through  April 

10, 1992. 

We  viewed  the  operation  of  CHCS  at  8  of  the  1 4  operational  test  sites  (12 
primary  test  sites  and  2  certification  sites):  Ireland  Army  Hospital,  Fort 
Knox,  Kentucky;  Naval  Hospital,  Charleston,  South  Carolina;  United  States 
Air  Force  Hospital,  Eglin  Air  Force  Base,  Florida;  Naval  Hospital, 
Jacksonville,  Florida;  98th  General  Army  Hospital,  Nuernberg,  Germany; 
United  States  Air  Force  Regional  Hospital,  Sheppard  Air  Force  Base, 
Wichita  Falls,  Texas;  Walter  Reed  Army  Medical  Center,  Washington,  D.C.; 
and  Malcolm  Grow  Medical  Center,  Andrews  Air  Force  Base,  Maryland. 

Fort  Knox  serves  as  an  alpha  test  site— a  site  where  initial  testing  for  new 
CHCS  software  is  conducted.  We  also  met  with  officials  of  SAIC  (the  prime 
contractor)  in  Falls  Church,  Virginia,  and  in  La  Jolla,  California;  officials  of 
the  four  Defense  OT&E  subcontractors-Arthur  D.  Little,  Inc.,  in 
Cambridge,  Massachusetts;  Vector  Research,  Inc.,  in  Falls  Church, 

Virginia;  Naval  Computer  and  Telecommunications  Station  in  Pensacola, 
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Florida;  and  MITRE  Corporation  in  Falls  Church,  Virginia;  officials  of  the 
CHCS  Program  Office  in  Falls  Church,  Virginia;  and  officials  of  Defense’s 
Program  Analysis  and  Evaluation  Office. 

We  worked  closely  with  senior  program  management  officials,  test 
officials,  and  OT&E  contractor  representatives  to  discuss  our  concerns  as 
they  arose,  and  confirm  our  understanding  of  potential  problems  and  their 
implications  for  the  achievement  of  test  objectives. 

We  conducted  our  evaluation  from  July  1 991  to  May  1992,  in  accordance 
with  generally  accepted  government  auditing  standards.  We  briefed  senior 
program  management  officials  during  our  review,  and  have  incorporated 
their  views  where  appropriate. 


I 
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Operational  Test  and  Evaluation 


Our  analysis  of  the  CT&E  results  showed  them  to  be  inconclusive  due  to 
inadequacies  in  both  the  test  plan  and  its  implementation.  To  determine  the 
effectiveness  and  suitability  of  CHCS,  Defense  established  five  test 
categories— deployment,  training,  mobilization,  technical,  and  functional. 
Defense’s  Office  of  Health  Systems  Evaluation  was  responsible  for 
managing  the  OT&E  process.  Defense  contractors  collected  and 
summarized  test  data,  and  Defense  appointed  and  trained  an  OT&E  analysis 
team,  made  up  of  15  members  representing  all  three  military  departments, 
to  review  and  evaluate  the  test  data.  The  Office  of  Health  S;  stems 
Evaluation  issued  an  OT&E  report  in  January  1992  identifiing  system 
strengths,  weaknesses,  and  deployment  risks  in  each  of  the  five  test 
categories. 

The  cues  program  manager  received  a  copy  of  the  OT&E  report  and  is 
developing  plans  to  address  the  system  weaknesses  identified.  We  also 
reviewed  the  OT&E  plans,  procedures,  and  results,  and  found  additional 
testing  inadequacies.  Specifics  on  these  inadequacies  are  discussed  below. 


Deployment  initially,  Defense  planned  to  conduct  OT&E  of  CHCS  deployment  at  13  sites, 

but  due  to  the  collection  of  unusable  or  incomplete  test  data,  changes  it 
made  in  its  overall  deployment  strategy,  and  congressional  concerns  about 
deploying  cues  to  additional  test  sites,  it  revised  the  test  plan  and  in  the 
end  tested  this  capability  at  only  one  site.  This  site  was  using  only  one  of 
the  nine  dies  functions— patient-appointment  scheduling.  The  results, 
therefore,  are  inconclusive  because  the  test  data  were  not  representative  of 
all  dies  functions  or  all  sites  to  which  deployment  is  intended.  The  Defense 
OT&E  team,  which  evaluated  the  deployment  capability,  found  the  data 
insufficient  for  evaluating  this  test  area  and,  as  a  consequence,  gave  it  no 
final  rating. 


Training  The  evaluation  of  the  effectiveness  and  suiUibility  of  user  training  for  CHCS 

required  te.sting  m  three  areas:  initial  training,  familiarization  training,  and 
continuing  training.'  In  1989,  when  Defense  began  deploying  dies  to  the 
original  te.st  sites,  it  also  began  collecting  data  for  use  in  its  OT&E  of  dies 
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training.  Defense  completed  the  OT&E  of  the  continuin;7-training  and 
familiarization-training  areas  at  each  of  three  representative  test  sites  that 
were  using  most  of  CHCS’  nine  functions.  However,  the  OT&E  for  the 
initial-training  area  proved  to  be  inadequate.  This  was  due  primarily  to  the 
fact  that  Defense  significantly  changed  its  procedures  for  conducting  initial 
CHCS  training  in  1991  and  re-tests  were  infeasible  since  personnel  who  had 
received  initial  training  were  no  longer  assigned  to  the  test  sites.  As  a 
result,  the  initial-training  OT&E  data  Defense  had  been  gathering  since  1989 
became  useless.  Since  none  of  the  original  CHCS  test  sites  was  suitable  as 
an  initial-training  test  site.  Defense  tested  this  capability  at  a  single,  newer 
test  site,  with  only  one  CHCS  function— patient-appointment  scheduling. 
However,  the  test  results  from  this  site  do  not  support  an  evaluation  of  the 
effectiveness  and  suitability  of  CHCS  initial  training  because  they  are  not 
representative  of  all  CHCS  functionality  and  sites  to  which  deployment  is 
intended. 


Mobilization  The  objective  of  testing  in  this  area  was  to  evaluate  CHCS’  effectiveness  in 

supporting  missions  encompassing  mass-casualty,  contingency,  and 
mobilization  situations  at  medical  treatment  facilities.  Testing  in  this  area 
involved  using  CHCS  during  Operation  Desert  Storm.  At  that  time,  however. 
Defense  had  not  integrated  CHCS  operations  into  its  command  mobilization 
and  contingency  plans.  Mobilization  testing  was,  therefore,  inadequate 
because  Defense  did  not  test  and  evaluate  CHCS  as  part  of  these  plans,  and 
Operation  Desert  Storm  was  completed  with  so  few  casualties  that  work 
loads  did  not  increase  sufficiently  for  sites  to  test  CHCS  performance  under 
stress. 

According  to  a  Defense  test  official,  although  plans  did  not  exist  at  the  time 
of  OT&E  for  using  CHCS  for  mass-casualty,  contingency,  and  mobilization 
situations.  Defense  has  now  prepared  such  plans.  Defense  does  not, 
however,  intend  to  lest  and  evaluate  these  plans  prior  to  its  1992 
deployment  decision. 


Technical  Defense  inadequately  implemented  OT&E  plans  for  several  technical  areas 

because  managers  did  not  control  test  conditions.  As  a  result,  some  tests 
v.’ere  not  conducted  in  a  realistic  operational  environment.  Defense  did  not 
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ensure  that  hospital  test  sites  had  activated  certain  system-security 
features.-  We  found  that  test  sites  had  deactivated  system-security  features 
to  alleviate  response-time  problems.  In  addition,  the  system-reliability 
computations  did  not  include  the  component  reliability  for  disk  drives.  As  a 
consequence,  system-hardware  reliability  test  results  are  inconclusive. 
Because  of  these  problems,  the  OT&E  did  not  adequately  evaluate  CHCS 
system  security  and  performance. 

A  test  official  stated  that  sites  had  not  activated  some  security  features 
because  Defense  had  not  issued  guidance  regarding  the  need  to  activate 
them  for  OT&E.  According  to  this  official.  Defense  is  attempting  to  address 
this  inadequacy  before  making  a  deployment  decision  by  (1)  requiring  the 
CHCS  program  office  to  certify  that  the  system  will  afford  medical  records 
the  required  confidential  level  of  security,  and  (2)  testing  and  evaluating 
CHCS’  audit-trail  security  feature  at  the  Charleston,  South  Carolina,  test 
site. 


Functional  Although  Defense  conducted  its  most  complete  testing  in  the  functional 

test  area,  we  found  weaknesses  in  Defense’s  data  collection  methodology 
relating  to  one  of  the  sbc  critical  operational  issues  in  the  functional 
area-quality  of  care.  Seven  of  the  27  tests  pertaining  to  this  issue 
contained  test-plan  or  test-execution  weaknesses.  These  weaknesses 
included  testing  at  a  small  number  of  nonrepresentative  sites  and  very  low 
response  rates  to  some  questions.  A  Defense  official  stated  that  these  data 
weaknesses  occurred  because:  (1)  only  a  few  test  sites  had  the  specific 
automated  capabilities  that  were  being  measured  during  OT&E;  (2) 
limitations  were  placed  on  the  number  of  survey  questions  to  minimize 
respondent  burden  and  encourage  more  accurate  responses;  (3)  test 
procedures  did  not  ensure  that  respondents  completed  survey  forms;  and 
(4)  data  were  not  comparable  because  those  questions  asked  after  CHCS 
installation  were  different  from  those  asked  before  CHCS  installation.  The 
above  weaknesses  impeded  any  meaningful  analysis  of  the  quality-of-care 
issue. 


“Test  sites  itiriied  uffatulil  trails  to  iireserve  eomputer  capaeity  and  protect  overall  system 
responsiveness  from  fiirtlier  deterioration 
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